Encoding information in illumination patterns

ABSTRACT

A method for encoding information in structured illumination pattern ( 120 ) includes selecting information to be encoded; characterizing a relief pattern ( 130 ) of an object&#39;s ( 115 ) surface; selecting at least one portion of the object&#39;s surface including the surface relief pattern; creating at least one structured illumination pattern responsive to the selected surface relief pattern and the selected information; and storing the structured illumination pattern.

CROSS REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly assigned, co-pending U.S. patentapplication Ser. No. ______ (Attorney Docket No. K000791), filedherewith, entitled ENDCODING INFORMATION IN ILLUMINATION PATTERNS, by P.Kane et al.; U.S. patent application Ser. No. 13/070,849, filed Mar. 24,2011, entitled DIGITAL 3D CAMERA USING PERIODIC ILLUMINATION, by P. Kaneet al.; and U.S. patent application Ser. No. 13/004/196, filed Jan. 11,2011, entitled FORMING 3D MODELS USING MULTIPLE RANGE IMAGES, by S.Wang; the disclosures of which are incorporated herein by reference.

FIELD OF THE INVENTION

The invention relates in general to encoding information and inparticular to using the three-dimensional surface characteristics forencoding information in structured light pattern.

BACKGROUND OF THE INVENTION

Automatic recognition and identification of objects, and embedding ofinformation in objects for this purpose, are well-known problems inelectronic imaging. Applications include industrial inspection andquality control, shipping and transportation, security and counterfeitdetection. In particular, the latter category, also referred to asobject authentication, is important to any industry dealing withvaluable or precision-made objects.

Common methods of embedding information that can be used to confirm theauthenticity of an object are signing, tagging or otherwise markingobjects of interest. For automatic purposes, a simple 1D or 2D (such asQuick Response or QR) barcode may be attached, stamped or engraved onobjects of interest. In another example, radio frequency identification(RFID) tags may be attached or embedded in the objects. In both cases,the added tags are encoded with identifying information that istransmitted to the recipients by some other means, and can be simplydecoded by well-known techniques to verify the identity of the sender.This is an important distinction, in that the above mentioned techniquesdo not, strictly speaking, verify the authenticity of the object. Thetags could in principle be placed onto unauthentic objects, or be fakedby other persons familiar with barcode or RFID technology, especiallybecause the presence of such tags is easily detectable. Also, mosttagging techniques require at least temporary modification of theobject, with tags that could be perceived as extraneous, altering objectappearance and aesthetics, which may be undesirable due to the size orvalue of the object.

In objects such as printed documents, banknotes, checks, etc.watermarking techniques can be used for embedding authenticatinginformation. In this case a special paper with the watermarks isrequired, which makes it more expensive. Because the watermarksthemselves are visible, they can be a target for counterfeitingattempts. Additional disadvantages of this approach are the facts thatwatermarks contain a very limited amount of information and cannot bechanged quickly.

Digital watermarking and steganography overcome the visibility ofembedded information, and rigidity issues such that the embeddedinformation can be decoded from the electronic file itself or afterscanning the document using a computer program. However there areseveral limitations of digital watermarking and steganography. Themethods can only be used for digital images and digital data, so theextraction of the information occurs via scanning printed documents oranalysis of electronic files and cannot be used on other physicalobjects. The amount of information in these methods is typically limitedto 32- or 64 bits. Extracting information in these methods requiresophisticated computational algorithms running on fast computers. If thedigital images or documents were resized or otherwise edited beforeprinting, embedded information can be distorted and consequentlyinaccessible. Digital watermarking and steganography methods arediscussed in the literature and in textbooks, for example in DigitalWatermarking and Steganography by Cox, Miller, Bloom, Fridrich andKalker, Morgan Kaufman, M A, 2008.

Other methods of identification and authentication rely on preciseoptical characterization of the object. Many non-contact,non-destructive methods have been suggested for optical scanning andrecognition of three-dimensional (3D) objects. In particular, methodsrelating to the present invention employ patterned illumination that isprojected onto the object. The unique interaction of the illuminationpattern with the object is then analyzed to determine the nature of theobject present. In one example, “Three-dimensional object recognition byFourier transform profilometry”, by Esteve-Taboada et. al., Appl. Opt.(38), 4760-4765 (1999) describes a method for recognizingthree-dimensional objects that combines the techniques of Fouriertransform profilometry (FTP) and the Joint Transform Correlator (JTC).In the FTP method, a periodic grating is projected onto a surface orobject using a projector, and its image (the reference image) isdetected by a camera. The recorded image is subjected to Fourieranalysis to determine the depth profile of the object.

In the JTC setup, the reference image is sent to a spatial lightmodulator (SLM), which is further placed inside a Fourier OpticalProcessor. The optical Fourier transform of the content displayed on theSLM appears at the output plane of the processor, which in turn isdetected by a second digital camera. For recognition, the referenceimage is present on the SLM, while the object to be tested is placedinto the FTP setup used to record the reference image. The image of thetest object is now also sent to the SLM inside the JTC, alongside thereference image. This results in the cross-correlation of the referenceobject and the test object to be output by the JTC, which can beanalyzed for recognition purposes by detecting correlation peaks at theoutput. Although useful, this technique has serious shortcomings whenconsidered for the authentication problem, which include: increasedspeed of processing at the expense of increased complexity of thehardware; the possession of the reference object is required; theproperties of the object shape are not taken into account in designingthe best projected pattern, which is restricted to a periodic grating;the method is not extensible to colored objects; for the method cannotbe easily extended to visual (human) authentication; and the method isnot easily extended to a sequence of objects.

U.S. Publication No. 2003/0231788 (Yuhkin et. al.) describes techniquesfor high-speed observation and recognition of an object that is within,or passes through, a designated area, using 3D image data and a varietyof 3D image capture techniques. The image capture techniques caninclude, but are not limited to, structured illumination. The method isbased on the generation of feature vectors, which must be compared to adatabase for recognition.

A method termed inverse fringe projection is described by Bothe et. al.in the paper “Compact 3D-Camera” (Proc. SPIE vol. 4778, 48-59, 2002).The image of the patterned illumination reflected from the object isrecorded by a digital camera. This image is mathematically inverted,such that when projected back at the object from the position of thecapture, it reflects from the object to re-create the originalillumination pattern at the site of the original projector, providedthat the original object is present in its original position andorientation. The authors describe the use of such a system inmanufacturing defect detection and quality control. Unlike previousmethods, the inverse fringe projection method takes into accountinformation about the shape of the reference object. The authors of thedescribe its use for compensation for the distortions of a projectedimage when projecting onto non-standard surfaces, such as brick walls orcorners in public spaces.

A similar technique is described in “Inverse Moire” by Jacques Harthongand Axel Becker, SPIE Vol. 3098, 1997, as a method of moiré metrology,where the shape of the object is measured by projecting a specific grid,inverse moiré, computed based on the knowledge of the object shape,allowing to analyze small deformations from a known mean shape withsimple fringe processing. This contrasts to the standard moiré metrologyapproach wherein a pattern of parallel straight lines is projected ontothe object surface, and the resulting pattern is analyzed using wellknown, though complex, fringe analysis techniques.

Structured light patterns are widely used for shape reconstruction, asdescribed for example, in “A state of the art in structured lightpatterns for surface profilometry”, by Salvi, Fernandez, Pribanic, andLlado, 2010, Pattern Recognition 43 (2010) 2666-2680. The process ofshape reconstruction using structured light is considered one of themost reliable techniques to recover object surfaces. To accomplish thisgoal, a calibrated projector-camera pair is used and, a light pattern isprojected onto the scene and imaged by the camera. Correspondencesbetween projected and recovered patterns are found and used to extract3D surface information. The projected pattern creates an illusion oftexture on the surface of an object, thereby increasing the number ofcorrespondences. They are chosen such as to uniquely codify each pixelposition in the image and consequently, on the object surface.

A variety of structured light patterns to enable discrete and continuouscodification for still and moving objects have been proposed. Theattributes of the patterns employed for 3D surface reconstruction are anumber of projected patterns, pixel depth, which is referred to thecolor and luminance level of projected pattern, periodicity of the setof patterns and others. Typical patterns consist of stripes (black andwhite or colored), sinusoidal gratings, luminance gradients withperiodic or fixed spatial or temporal structure, and others. What isimportant, in all these cases patterns are used to obtaincorrespondences between object surface locations and pixels in thecaptured images, where subsequently the depth map of the object surfaceis reconstructed from pattern deformations using ray tracing andtriangulation techniques. Unlike inverse fringe projection techniques,these patterns are not adapted to reflect 3D properties of a particularstatic or moving object, but are used as a means to reconstruct depthmaps of object surfaces.

While these approaches take into consideration 3D surface information indesigning projection patterns, they cannot be directly applied forauthentication problems that are based on encoding information utilizingsurface, embedding information into a surface and retrieving thisinformation as a means of authentication for a variety of productsincluding printed documents. Therefore, there is a need for methods toauthenticate an object based on an automatic, noninvasive examination ofits features and embedded information that can be concealed and modifiedas required.

SUMMARY OF THE INVENTION

Briefly, according to one aspect of the present invention a method forencoding information in structured illumination pattern includesselecting information to be encoded; characterizing a relief pattern ofan object's surface; selecting at least one portion of the object'ssurface including the surface relief pattern; creating at least onestructured illumination pattern responsive to the selected surfacerelief pattern and the selected information; and storing the structuredillumination pattern.

The invention and its objects and advantages will become more apparentin the detailed description of the preferred embodiment presented below.

BRIEF DESCRIPTION OF THE DRAWINGS

While the specification concludes with claims particularly pointing outand distinctly claiming the subject matter of the present invention, itis believed that the invention will be better understood from thefollowing description when taken in conjunction with the accompanyingdrawings, wherein:

FIG. 1 is a flow diagram of a method for encoding information in astructured illumination pattern.

FIG. 2 is an example of the method for characterizing a relief surfaceof the object using structured light system.

FIG. 3 is an example of the critical surface characteristics.

FIGS. 4 a and 4 b are examples of the information selected to be encodedin the illumination pattern.

FIG. 5 is an example of the structured illumination pattern with encodedinformation applied to the critical surface.

FIG. 6 is an example of creating the structured illumination patternusing inverse projection method.

FIG. 7 is an illustration of the reflected pattern used forauthentication process.

FIG. 8 is a process diagram for authenticating the object using encodedinformation.

FIG. 9 is an example of a different usage of encoded informationillustrated as a process of accessing a database via retrieving encodedinformation contained in the object surface using structuredillumination pattern.

FIG. 10 is a flow diagram of a method for authenticating an object usinginformation encoded in a structured illumination pattern, in accord withthe present invention.

FIG. 11 is a flow diagram of a method for authentication ofthree-dimensional structures, in accord with the present invention.

FIG. 12 is a flow diagram of a method for authentication ofthree-dimensional structures, in accord with the present invention.

FIG. 13 is a flow diagram of an alternative method for authentication ofthree-dimensional structures, in accord with the present invention.

FIG. 14 is a flow diagram of another embodiment of the method forauthenticating an object using information encoded in a structuredillumination pattern, in accord with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention will be directed in particular to elements formingpart of, or in cooperation more directly with the apparatus inaccordance with the present invention. It is to be understood thatelements not specifically shown or described may take various forms wellknown to those skilled in the art.

In order to ensure a robust encoding, and data retrieval that areapplicable for a range of objects and items, as well as usages such asauthentication, the preferred embodiment of the present inventionfocuses on three key elements: information to be encoded, a specifiedsurface texture that is identified, created or modified on the surfaceof the object or an item, and which is used in the process of encoding,retrieving and/or decoding information, and structured light pattern,which carries encoded information and is applied on the specifiedsurface texture in order to retrieve and/or decode the information. Allthree elements are specified, designed and utilized in a complimentaryfashion, by taking into consideration their respective characteristicsand properties in the process of encoding.

In the context of the present invention we understand the terminformation to mean images, text, characters, numbers, codes in variousformats, symbols, data or geometrical patterns. Such information can berelated to or carry a certain meaning or association with respect to theobject or an item, for example, in the form of a tag or metadata orother data relevant to the object or object characteristics or usage.Examples of such relevant information include time and date of theobject manufacturing, a company that produced the object, dataauthenticating the object, information about the object's usage andspecifications, historical information about the object, informationabout the object's owner, or a World Wide Web address to databases withthe description and knowledge about the object. Alternatively, theinformation can be unrelated to the object.

Examples of this type of information include but are not limited to amap of the local area, or a message to a person, or a group, enabled todecode it. In any case the object and object's surface texture are usedtogether with the structured light to encode and decode information.

Encoding is defined as the conversion of information from one system ofcommunication or representation into another, and especially, theconversion of a message or other type of information into code. Toaccess encoded information the inverse process of decoding is required,wherein the specifics of the decoding depends on the details of theencoding process and can include access to a key. A key is a piece ofinformation or a parameter that determines the output of an encoding ordecoding method, process or algorithm.

By structured light or structured illumination it is meant that apattern of projected visible or non-visible radiation. The termsstructured light, structured light pattern, structured illumination, andstructured illumination pattern are used interchangeably in the contextof the present invention.

By the surface texture it is meant that the surface or a specificportion of the surface of the object or item, which is employed in theprocess of encoding and retrieval or decoding information, and which hasan associated three-dimensional or relief pattern.

The relief characteristics of the object surface or a specific portionof the surface, are used for information retrieval. Surfacecharacteristics include, but are not limited to, physical dimensions ofthe three-dimensional relief, geometrical arrangement of patternelements, and size or scale of relief patterns. Relief characteristicsalso include material properties of the surface texture, characterizedby their reflectance, transmittance and absorbance of light of differentwavelengths, which together with the relief characteristics can uniquelyidentify the object, a type of objects or a family of objects.

These properties of the surface texture are specifically designed,selected, determined or created, to be unobvious and unrecognizablewithout an inspection mechanism. The surface or portion of the surfacethat is used in the process of encoding information or data or metadataor tags can be called critical surface texture, critical surface texturepattern, critical surface pattern, or, simply, critical surface. Indescribing these types of surface properties, we can also use the termssuch as critical surface (or texture or relief) features and elements.

In some embodiments, the surface texture used for encoding and decodinginformation into and from a structured illumination pattern isspecifically selected or designed at the stage of producing the object.This way the identified surface becomes an intrinsic part of the objectitself. Any physical 3D object having a surface texture whichdifferentiates it from similar objects can be utilized in an encodingprocess provided that the relief characteristics are within thespecifications of the process.

In other embodiments, the texture can be added to the object at a laterstage, therefore requiring a purposeful object modification, forexample, while printing the document or post-manufacturing retouching ofthe object. However, unlike in the cases of a barcode or an RFID tag,the texture is added at a micro or intermediate scale levels, ratherthan a macro scale level, without disruptive change of the objectappearance, thus, effectively becoming a part of the object. A printedimage or a document with the 3D texture pattern deposited or embossed onthe print surface is an example of such an object.

The information retrieval or decoding mechanism employed in the presentinvention is structured light or structured illumination. A pattern ofthe structured light or structured illumination is projected onto thecritical surface texture. The structured illumination pattern isco-designed, selected, determined or created with the critical surfacetexture, and carries encoded information. The critical surface textureand the structured illumination pattern, can be considered to act as alock and a key in the encoding and information retrieval process. Boththe structured illumination pattern and the critical surface texture areneeded to encode and decode the information that is used for, tagging,digital watermarking, steganography, and other applications. Thestructured illumination pattern is specifically designed such that itsapplication will result in accessing and retrieving the encodedinformation.

The designed structured illumination pattern also takes intoconsideration the properties of the information to be encoded orretrieved, and is scrambled or distorted with respect to the targetinformation. the pattern is constructed such, that the final outputinformation or output signal, that is retrieved cannot be easilyaccessed, recognized or guessed from the structured illumination patternitself, without reference to the surface texture used to create it. Inthe preferred embodiment of the present invention the encodedinformation can be text, characters, numbers, images, or other types ofinformation such as symbols or geometrical configurations. Theinformation itself can be selected or defined in conjunction with thesurface properties and structured illumination. Alternatively,information can also be represented as codes, encoded in various waysand forms, such that when this code or codes are retrieved they mayfurther be processed and decoded if desired, similarly to digitalwatermarking and steganography applications. In the context of thepresent invention it is useful to differentiate two aspects ofinformation: its content or meaning of what is being communicated (forexample, the meaning of a word or a text message), and the physical formin which information is represented (for example, printed charactershaving particular size, thickness or color). The structured illuminationpattern can be monochromatic or polychromatic, and can include spectraof the wavelengths beyond visible, for example, infrared or ultraviolet.The inclusion of non-visible wavelengths may be important when thecritical surface texture contains features and elements that selectivelyabsorb or reflect non-visible wavelengths, or that, phosphoresce orfluoresce responsive to the structured illumination. This enhances theability to encode data using structured illumination, as well asincrease the sophistication of the encoded information.

The structured illumination pattern and the information containedtherein can be described in terms of complexity, entropy, spectralcontent, spatial resolution, size, and or other parameters orcharacteristics known in the art. The structured illumination patternscan be stored as digital data files, digital image files or digitalvideo files. In the latter embodiment the structured illumination isorganized as a series of patterns sequentially projected in frames ofthe video file onto the object surface following a prescribed order.Alternatively, a set of structured light patterns can be combined in acomposite image for projection, where individual parts of the compositedstructured illumination pattern constitute separate frames stored as avideo file. Encoded information in such a case is organized in such away, that individual portions of information are retrieved in a sequenceof steps as separate frames, which can be observed and/or captured usingimaging devices including digital still and video cameras, spectrometersand spectrophotometers, mobile devices equipped with the digital stilland video cameras and other imaging mechanisms, and different types ofscanners depending on the geometrical and material properties of theobject and surface of interest. Subsequently, such individual frames,parts or portions of obtained information can be optionally combined,stored as digital files and processed in a number of ways, such ascreating video, or image sequence, or by overlaying, compositing,averaging, subtracting and/or using various other operations andtechniques typically employed in image processing and analysis. In theembodiment, wherein the structured illumination pattern or patterns arestored as digital data files in a computer memory, the stored data canbe used to reconstruct illumination patterns.

In one configuration, the retrieved information is used forauthenticating the object. The authentication includes comparing theretrieved information with stored data pertaining to the authenticreference object. In other words, the authentication is based on whetherthere exists or does not exist a match between stored data pertaining tothe known reference authentic object and the information retrieved fromthe object under consideration or inspection.

If the retrieved information requires additional decoding, knowndecoding and cryptographic techniques can be used to retrieve and decodethe data. For example, tables, alphabets, or searching methods can beapplied to associate codes with online databases and Internet websites,via IP addresses, URL or passwords, where more information can be found.Alternatively, encoded information can be also used for other purposes,such as providing data related to the object, such as the dimensions,composition, content, the owner or manufacturer of the object, usageinstructions, including storage characteristics, environment, safetyprecautions, etc. In other embodiments, information can contain dataunrelated to objects, such as secret messages, instructions, historydata about place, people, and other types of information.

The present invention can be implemented in computerized equipment andcomputer hardware. For example, the method can be performed using acomputerized system including one or more digital cameras or othercapture devices, one or more digital projectors, and one or morepersonal computers or other suitable electronic devices, wherein thedevices have the capability to store and project structured illuminationpatterns and images, and capture images of reflected patterns afterprojection, store and analyze these captured image, and transmit theimages wirelessly, or via connection to online Internet or Ethernetsites or computerized databases.

FIG. 1 shows a schematic of a system for encoding information in astructured illumination pattern, in accordance with the presentinvention. A projector 110 is used to project an illumination pattern120 onto an object 115 from a projection direction 315. The illuminationpattern 120 can include a single still image or pattern, a sequence ofstill images or patterns, a video sequence, or a series of videosequences. An image of the object 115 is captured using a digital camera140 from a capture direction 335. The capture direction 335 can bedifferent from the projection direction 315. In this case the projector110 and digital camera 140 are separated horizontally along a baseline226 in order to provide depth information according to the parallaxeffect. A sequence of one or more of different illumination patterns 120is projected onto a selected portion 135 of the object's surface thatincludes a relief pattern 130. The camera 140 captures an imagecorresponding to each of the projected illumination patterns.

The phrase “digital camera” is intended to include any device includinga lens which forms a focused image of a scene at an image plane, whereinan electronic image sensor is located at the image plane for thepurposes of recording and digitizing the image. These include a digitalcamera, cellular phone, digital video camera, surveillance camera, webcamera, television camera, electronic display screen, tablet or laptopcomputer, video game sensors, multimedia device, or any other device forrecording images.

The phrase “processor-accessible memory” is intended to include anyprocessor-accessible data storage device, whether volatile ornonvolatile, electronic, magnetic, optical, or otherwise, including butnot limited to, registers, floppy disks, hard disks, compact discs,DVDs, flash memories, ROMs, and RAMs.

The phrase “communicatively connected” is intended to include any typeof connection, whether wired or wireless, between devices, dataprocessors, or programs in which data is communicated. Further, thephrase “communicatively connected” is intended to include a connectionbetween devices or programs within a single data processor, a connectionbetween devices or programs located in different data processors, and aconnection between devices not located in data processors at all.

The projector includes a light modulator 220, which is a digitallyaddressed, pixelated array such as a reflective LCD, LCoS, or TexasInstruments DLP™ device, or a scanning engine, which is projected ontothe scene by a projection lens 210. Many illumination systems for suchmodulators are known in the art and are used in conjunction with suchdevices. FIG. 1 shows an illumination system 225 that illuminates themodulator 220. The illumination system 225 may include sources such asincandescent lamps, light emitting diodes (LEDs), lasers or laser diodeswith associated collimating or condensing optics. The structuredlighting system comprised of the capture lens 205, image sensor 215,projection lens 210 and light modulator 220 can operate in visible ornon-visible light. In one configuration, near-infrared illumination isused to illuminate the scene objects, which is less distracting topeople who are in the scene, provided that the intensity is kept at safelevels. Use of infrared wavelengths is advantageous because of thenative sensitivity of silicon based detectors at such wavelengths.

The projector can also be included within an electronic device, such asa digital camera. The Nikon COOLPIX S1200pj digital camera is oneexample of a digital camera that includes a built-in projector. Inanother configuration, the projector can be built into a mobile phone orother personal electronic device, such as a tablet or notebook computer,PDA or other suitable device.

The projector is communicatively connected to a processor 230, which canbe an integral part of the projector, be a part of a larger device,which has the projector as its part, or, alternatively, the projectorcan be connected wirelessly or through the wired connection to aseparate computer (not shown) having a processor.

The camera 140 is communicatively connected to the processor 230 thatcommunicates with the image sensors 215, and light modulator 220. Thecamera 140 further communicates with a user interface system 245, and aprocessor-accessible memory system 250 via the processor 230, which iscommunicatively connected to the processor-accessible memory system 250and the user interface system 245.

Although the processor-accessible memory system 250 is shown separatelyfrom the processor 230, one skilled in the art will appreciate that itis possible to store the processor-accessible memory system 250completely or partially within the processor 230. Furthermore, althoughit is shown separately from the processor 230, one skilled in the artwill appreciate that it is also possible to store the user interfacesystem 245 completely or partially within the processor 230.

Through the processor 230, the processor-accessible memory system 250and the user interface system 245 are also communicatively connected tothe projector 110.

In one configuration, such as the one shown in FIG. 1, the processor230, memory 250 and the user interface 245 are separate from the camera140 and projector 110. In this case, the processor 230, the memory 250and the user interface 245 can be a part of one or more separatedevices, such as, for example, a personal computer or other suitablecomputing devices. In other configurations, a memory 250 and userinterface 245 can also be located within or on the camera 140, and aseparate memory 250 and user interface 245 can also be located within oron the projector 110. In yet still other configurations, all componentsincluding the projector 110, the camera 140, the processor 230, thememory 250 and the user interface 245 can be integrated in one hardwaresystem.

The processor 230 can include one or more data processing devices thatimplement the processes of the various embodiments of the presentinvention. The phrases “data processing device” or “data processor” areintended to include any data processing device, such as a centralprocessing unit (CPU), a desktop computer, a laptop computer, amainframe computer, a personal digital assistant, a Blackberry™, adigital camera, cellular phone, or any other device for processing data,managing data, or handling data, whether implemented with electrical,magnetic, optical, biological components, or otherwise.

The processor-accessible memory system 250 includes one or moreprocessor-accessible memories configured to store information, includingthe information needed to execute the processes of the variousembodiments of the present invention. In some configurations, theprocessor-accessible memory system 250 is a distributedprocessor-accessible memory system including multipleprocessor-accessible memories communicatively connected to the processor230 via a plurality of computers or devices. In some configurations, theprocessor-accessible memory system 250 includes one or moreprocessor-accessible memories located within a single data processor ordevice.

In this regard, although the processor-accessible memory system 250 isshown separately from the processor 230, one skilled in the art willappreciate that it is possible to store the processor-accessible memorysystem 250 completely or partially within the processor 230.Furthermore, although it is shown separately from the processor 230, oneskilled in the art will appreciate that it is also possible to store theuser interface system 245 completely or partially within the processor230.

The user interface system 245 can include a touch screen, switches,keyboard, computer, or any device or combination of devices from whichdata is input to the processor 230. The user interface system 245 alsocan include a display device, a processor-accessible memory, or anydevice or combination of devices to which data is output by theprocessor 230. In this regard, if the user interface system 245 includesa processor-accessible memory, such memory can be part of theprocessor-accessible memory system 250 even though the user interfacesystem 245 and the processor-accessible memory system 250 are shownseparately in FIG. 1.

In an alternate configuration shown in FIG. 2, two digital cameras 140Aand 140B are shown separated by a horizontal baseline 226, on alternatesides of the projector 110, which projects a structured light pattern120A onto the object 115. The two cameras 140A and 140B include capturelenses 205A and 205B, and image sensors 215A and 215B, and both camerasare communicatively connected to the processor 230, the user interfacesystem 245, and the processor-accessible memory system 250. Using twocameras with matched optical and electronic characteristics, inconjunction with the different perspectives of the two cameras relativeto the position of the projector 110, provides additional depthinformation pertaining to the surface texture 130 according to theparallax effect.

As described earlier, the illumination system 225 can operate in visibleor non-visible light. Similarly, the digital cameras 140A and 140B canhave spectral sensitivities that are tuned to visible or non-visiblelight, and can be tuned to matching regions of the spectrum, partiallyoverlapping regions of the spectrum, or non-overlapping regions of thespectrum. In FIG. 2, an additional illumination pattern 120B is shown,represented in the figure by a different line style. In thisconfiguration, pattern 120A is created by the projector 110 and isdetected by camera 140A, while pattern 120B is created by the projector110 and is detected by camera 140B. Illumination patterns 120A and 120Bmay be projected simultaneously or in sequence. Depending on thespectral content of patterns 120A and 120B, and the spectralsensitivities of digital cameras 140A and 140B, the two cameras mayrecord all, part or none of the reflected energy from each of theilluminating patterns. In this manner, the surface texture 130 can becharacterized simultaneously, or in succession, in different regions ofthe spectrum. In some configurations, the cameras 140A and 140B havematching optical and electronic characteristics, such as sensorresolution, f-number of the camera lens, and focal length, while inother configurations these are mismatched.

Images captured with the camera 140 and illumination patterns 120 ofFIG. 1 or illumination patterns 120A and 120B and images captured withthe cameras 140A of FIG. 2, can be stored in the form of files of dataor files of digital images. A digital data file refers to any computerfile, that is, a file maintained in computer-readable form. Such filescan contain any type of data, for example, a word processing document orspreadsheet, image or video data. An image data file pertains to filescontaining information on digital images.

A digital image includes one or more digital image channels, digitalimage bands or color components. Each digital image channel is atwo-dimensional array of pixels. Each pixel value relates to the amountof light received by the imaging capture device or projected by aprojector device corresponding to the physical region of pixel. Fortypical capture and projection devices, a digital image will oftenconsist of red, green, and blue digital image channels. When morespectral bands are utilized for capture and projection, the digitalimages files may contain many bands wherein each band corresponding to acertain regions of captured and or projected spectral light. Video ormoving images can be thought of as a sequence of digital images.Although a digital image channel is described as a two dimensional arrayof pixel values arranged by rows and columns, those skilled in the artwill recognize that the present invention can be applied to nonrectilinear arrays with equal effect. Images are stored in theprocessor-accessible memory 250. The processor-accessible memory canalso store a computer program in a form of software that can be requiredto project, capture, process and analyze digital data and or digitalimages.

It should also be noted that the present invention can be implemented ina combination of software and hardware and is not limited to devices,which are physically connected and/or located within the same physicallocation. One or more of the devices and parts illustrated in FIG. 1 andFIG. 2 can be located remotely and can be connected via a network. Oneor more of the devices can be connected wirelessly, such as by aradio-frequency link, either directly or via a network.

FIG. 3 shows a process diagram for a method of creating an encodedillumination pattern in accord with the present invention. Beginningwith step 310, the surface relief of the object is characterized.Characterizing a relief pattern of an object's surface includes forminga 3D model of the relief pattern using the system of FIG. 1 or FIG. 2.In a preferred configuration, this is accomplished using a structuredlighting technique such as those described in commonly assigned,co-pending U.S. patent application Ser. Nos. 13/070,849 (and 13/004,196(hereafter, '196). Returning to FIG. 1, a sequence of patterns 120 areprojected using the projector 110 onto the object 115. The sequence ofpatterns 120 can include, but is not limited to, spatially periodicbinary patterns such as Ronchi Rulings or square wave gratings, periodicgray scale patterns such as sine waves or triangle (saw-tooth)waveforms, or dot patterns.

In a preferred configuration described in the '196 application, thesequence of patterns 120 includes both spatially periodic binary andgrayscale patterns, wherein the set of periodic grayscale patterns eachhas the same frequency and a different phase, the phase of the grayscaleillumination patterns each having a known relationship to the binaryillumination patterns. The sequence of binary illumination patterns isfirst projected onto the scene, followed by the sequence of periodicgrayscale illumination patterns. The projected binary illuminationpatterns and periodic grayscale illumination patterns share a commoncoordinate system having a projected x coordinate and a projected ycoordinate, the projected binary illumination patterns and periodicgrayscale illumination patterns varying with the projected x coordinateand being constant with the projected y coordinate. In this preferredconfiguration, the sequence of binary patterns are the well-known graycode patterns, and these are used to obtain a coarse estimate of thelocations in the projected illumination patterns 120 that correspond tothe pixel locations in the images of the binary patterns captured by thedigital camera 140 on the image sensor 215. The sequence of grayscalepatterns is a set of three sinusoidal intensity patterns that areidentical, except that the second and third patterns are shifted inphase by ⅓ and ⅔ of a period relative to the first pattern. Furthermore,in this preferred embodiment, the phase of the projected sinusoidalpatterns is arranged such that the maximum of one of the patterns (i.e.crest of the waveform) is aligned with the transitions between theon/off regions in the gray code patterns. In this way, the sinusoidalpatterns can be used to refine the estimates of the locations in theprojected illumination patterns 120 that correspond to the pixellocations in the images of the binary patterns captured by the digitalcamera 140 on the image sensor 215, filling in the values between thecoarse estimates. Full details of this method are provided in the '196application.

Once the mapping from projector to camera is established, methods knownin the art can be used to compute range or depth values that correspondto a 3D characterization or mapping of the relief pattern 130. The '196application describes the details of this process. Such 3Dcharacterization of the surface texture in the form of thethree-dimensional model of the object's surface is stored in the memory250.

Additionally, other information related to the surface texture, such asspectral and material properties can be acquired and stored in theprocessor accessible memory 250 as a part of the model or separately.Spectral information can be acquired using structured light system ofFIG. 1 or FIG. 2, wherein the projected light patterns 120 can beadapted to include patterns of specifically selected wavelengths,including the non visible parts of the spectrum. Alternatively, spectralproperties of the object surface can be measured using commerciallyavailable devices such as for example, the PR-655 SpectraScanSpectroradiometer sold by Photo Research, Inc. Information aboutmaterial properties can be obtained from the manufacturer or via specialanalysis techniques typically employed in analytical science. Thesemethods analyze small quantities of objects' material substance byapplying techniques of mass spectroscopy, gas chromatography or otheradvanced analytical methods. In another embodiment, the characterizationof the surface relief can be pre-determined in terms of themanufacturing specifications and available at the step 310 in the formof a data file. In yet other embodiments, techniques of profilometry,elipsometry or other tools and methods can be applied to acquire theobject's three-dimensional surface relief characterization.

FIG. 4 a shows an example of a surface with a relief pattern that can beused for information encoding, in accord with the present invention.FIG. 4 a shows a schematic of a substrate 400 with an array of surfacerelief elements, in particular relief elements 405, 410, and 415, eachelement composed of a series of concentric, raised rings. All dimensionsin the figure are given in inches. The upper left corner of FIG. 4 ashows a cross sectional view of elements 405 and 410 that indicates theheight of the rings above the substrate. FIG. 4 b shows a binary imagerepresentation 420 of the surface of FIG. 4 a, where the pixelbrightness corresponds to height above the substrate. That is, whitepixels 425 represent maximum surface height, and black pixels 430represent zero surface height above the substrate. A binary image suchas 420 can be stored in computer readable memory as a digital imagefile, and sent to a writing device to fabricate a textured surface.

Returning to FIG. 3, in step 312, the information for encoding isselected. The selection process is based on the specific application forthe encoding, as well as surface relief characteristics. For example, ifthe encoding is used for authentication of the object, and the selectedcritical surface texture contains surface relief characteristics thatuniquely identify and differentiate the object from similar objects, theinformation for encoding can be selected such that its content takesinto consideration the manufacturer of the object, date and time of theobject creation, its important usage instructions, etc. At the sametime, the physical embodiment of the information is chosen to correspondto the critical surface characteristics, such as, for example, itsgeometrical and material properties, including the surface depthvariation, spatial frequency of the surface 3D features and or itsreflectance or absorption spectra. It is important to recognize that thegeometrical and material properties of the surface relief pattern caninfluence the selection of how information is physically represented. Itstems from the desirability of a requirement that the output encodedinformation satisfies a specific criterion, for example, how easily theinformation can be accessed and decoded from its encoded form. In thepreferred embodiment of the invention, the selected information,especially, its physical representation is chosen in step 312 to satisfythe specific criterion of the amount of distortion that can be createdwhen the original information in the form of a pattern of illuminationis projected onto the surface texture. In other embodiments selectioncriteria can also include spectral composition, text language, visualcomplexity and other relevant criteria which can be linked to suchconsiderations as whether the information will be retrieved manually, orautomatically, by visual inspection of the projected illuminationpattern, or by computational analysis of the captured projection of thepattern, and other relevant considerations.

FIG. 5 illustrates one embodiment of a method for selecting theinformation to be encoded, in accord with the present invention.Specifically, this example illustrates how physical characteristics ofthe information can be selected with specific unrecognizability orillegibility criterion when the content of information is a textmessage. In general, when the font size is large in comparison with theparameters or features of the surface elements the amount of apparentdistortion created by the surface is small. As the font size is reduced,the amount of distortion introduced by the surface relief patternincreases. FIG. 5 shows a photograph of a critical surface texture 500,consisting of a substrate with an array of surface relief elements,similar to the schematic of FIG. 4 a. The surface texture 500 has auniformly diffuse white coating. Projected onto the rows of surfacerelief elements are rows of characters of varying font size. Row 505 isthe largest row, and shows very little distortion by the surface relief.The next row 510 is smaller, and while it shows more distortion canstill be easily read. The next row 515 is smaller yet, but can be readwith some difficulty. The smallest rows, 520 and 525 is yet smaller, andare very difficult to read. Note that rows 520 and 525, have a font sizeon the order of the feature size of the individual surface reliefelements.

The amount of distortion introduced depends on the relative geometricalparameters of the surface texture, and the physical characteristics ofthe information. In the preferred embodiment, the amount of distortioncan be assessed visually. In such an embodiment, a distortion visibilitythreshold can be specified such that the threshold is exceeded, theinformation is unrecognizable. The distortion visibility threshold canbe directly related to the underlying parameters of the information,such as the font size, that can in turn be used to directly specify thedistortion visibility threshold.

Alternatively, the amount of distortion can be assessed using methods ofcomputer vision, where the captured distorted information is analyzedwith the goal to retrieve and or recognize information. In such a case,typical performance characteristics describing computer vision and imageanalysis algorithms can be evaluated. For example, precision and recallare accessed and used in order to satisfy an unrecognizabilitycriterion, and consequently select information for encoding.Corresponding thresholds related to the distortion and underlyingphysical characteristics of the information, can also be established.Alternatively, information can be selected via the process of modelingthe potential distortion introduced by the specified surfacecharacteristics and using a pre-established threshold.

Returning now to FIG. 3, in step 320, one or more specific portions ofthe surface relief pattern in selected. In this step, a process similarto the process described in step 315 can be employed. In particular, aspecific portion of the surface relief pattern can be selected toprovide a sufficient degree or amount of distortion introduced by theportion of the surface relief pattern to the information. To this endvarious candidate portions of the surface relief pattern are evaluatedby assessing the amount of distortion corresponding to each portion ofthe surface. Next, the portions are compared with respect to thecorresponding amount of distortion and the portion with the highestamount is chosen. In the preferred embodiment, a visual evaluation canbe performed. In alternative embodiments, model-based evaluation orautomatic evaluation using captured images of projected information ondifferent surface relief portions can be used. In other embodiments, thesurface portion selection criteria can include, for example, thepresence of unique surface characteristics representative of the object,which are contained in the portion under selection consideration,including spectral, material and geometrical characteristics of thesurface texture or surface relief pattern.

Information can be divided into parts, fragments or frames.Consequently, several portions of the surface texture can be selectedfor the encoding, wherein different selected portions can be used forencoding parts or fragments of information. For example, if theinformation consists of several frames of digital images or several datafiles, each piece of information can be associated with one or moreselected portions of object's surface texture with the correspondingrelief pattern. In such embodiment each of the portions can be selectedby applying different criteria and different evaluation methods, whichare the most appropriate for corresponding parts of information.

Returning to FIG. 3, the next step 325 involves creation or constructionof one or more illumination patterns with encoded information responsiveto the selected sections of the surface. In the preferred embodimentthis step can be accomplished using a method such as an inverseprojection method. Accordingly, within this step 325, using structuredlighting techniques, first the mapping between projector coordinates andcamera coordinates is determined. This mapping can be obtained directlyfrom the methods described in the '196 patent application. Conceptually,the forward mapping is represented by a pair of two-dimensionalfunctions u(x,y) and v(x,y), where x,y are spatial coordinates in thelight modulator plane 220, and u,v are spatial coordinates in the imagesensor plane 215. The functions u and v contain information about thethree-dimensional object 115 and its textured surface 130, and are amapping of the light rays leaving the modulator at x,y and arriving atthe sensor at u,v. In practice, sampled functions u_(s)(x_(i),y_(j)) andv_(s)(x_(i),y_(j)) are measured at discrete pixel locations x_(i),y_(j)and u_(n),v_(m) in the corresponding planes. The sampled functionsu_(s)(x_(i),y_(j)) and v_(s)(x_(i),y_(j)) are simply mappings betweenthe columns and rows of pixels in the two planes. For completeness, thegrayscale responses of the projector 110 and the digital camera 140 mustalso be measured.

Once the forward mappings u(x,y) and v(x,y) (or their sampledcounterparts u_(s)(x_(i),y_(j)) and v_(s)(x_(i),y_(j))) are known, forany image I_(proj)(x,y) at the projector we can calculate the resultingimage I_(cam)(u,v) at the camera, which is a distorted or warped versionof I_(proj)(x,y), after reflection from the surface texture 130. It isthis image I_(cam)(u,v) which can be used as the starting point forinformation encoding. To encode information using the surface texture130, the inverse transformations X(u,v) and Y(u,v) are computed.Physically, these transformations describe, for a given location u,v inthe image sensor plane, the corresponding location in the projectorplane x,y that illuminates the location u,v. In practice, discrete(sampled) versions of these transformations are computed. To simplifythe notation, will refer to X(u,v) and Y(u,v), with the understandingthat in practice the quantities u,v and X,Y are discrete rather thancontinuous.

A variety of methods may be used to compute the inverse transformationsX(u,v) and Y(u,v), depending on the complexity of the surface texture130. One method is proposed by W. Li et. al. in Object adapted patternprojection—Part I: generation of inverse patterns, published in Opticsand Lasers in Engineering vol. 41, pp. 31-50 (2004). This method worksin the projector space (X,Y), where the data are on a rectangular grid,and sequentially processes neighborhoods of four projector pixels.Numerical values for the four pixels are interpolated from thesurrounding camera pixels, which can lie on a non-rectangular grid whenviewed from the projector space. The interpolation coefficients arerecomputed for each neighborhood, and the interpolation is performedusing a linear model.

Once the inverse transformations X(u,v) and Y(u,v) are known, it is thenpossible, given an image I_(cam)(u,v) at the camera to compute thecorresponding image I_(proj)(x,y) that must be input to the projector toresult in the image I_(cam)(u,v) at the camera. Therefore, given thesurface texture 130, and the inverse transformations X(u,v) and Y(u,v),the information to be encoded is specified as the image I_(cam)(u,v).The encoded information is then embedded in the corresponding imageI_(proj)(x,y). The image I_(proj)(x,y) may be regarded as a structuredillumination pattern with encoded information. Note that the forwardtransformations u(x,y) and v(x,y), and the inverse transformationsX(u,v) and Y(u,v), are specific to the digital projector and the digitalcamera used, as well as the critical surface texture. This dependenceincludes the relative positions of these devices, in terms of theirdistances and angles, with respect to the surface. Changes in any ofthese parameters will affect the retrieval of information from thesystem. In addition, changes in these parameters will require arecomputation of the forward and inverse transformations.

Other ways to create encoded illumination patterns responsive to theselected sections of the surface can utilize modeling and simulationtechniques using computer means, including computer graphics techniquesthat are widely utilized in 3D graphics design and programming forcomputer gaming, animation and CAD systems. For example, the surfacemodels and ray tracing techniques can be used to simulate an 3Dinversion of the surface relief pattern and projection of the selectedinformation in the form of an image onto the modeled inverted surface.Recorded simulated reflection via computing inverted selected area ofthe surface texture with the surface relief pattern, and recording thesimulated reflection of the simulated or visualized projection. Sincethe selected portion or portions of the surface texture with the reliefwere chosen such as produce a distortion.

Additionally, spectral properties of the structured light pattern may bechosen such that they selectively interact with absorption, transmissionand reflectance properties of the surface relief pattern. For example,the structured light patterns can be created to include colors thatmatch or are complimentary to the colors of the surface relief pattern.If these patterns are projected onto the surface, color information inthe reflected pattern will be modified: in can become more vivid ifcolors match, or will appear achromatic if colors are complimentary.Alternatively, the structured illumination pattern can be designed tostimulate the emission of radiation from the object's surface, when theobject's surface material can fluoresce or phosphoresce.

Once illumination patterns are created in step 325 responsive to theselected surface relief patterns that contain encoded information, instep 330 they are stored in the memory of the computer or othercomputerized equipment or transmitted to the end user for storage andusage in according to the intended application. The process of FIG. 3described a method of encoding information using an existing object withthe native surface relief pattern. Alternatively, the process can startwith creating a relief pattern to use during the encoding process in theform of a model or a prototype, and followed by identifying, modifyingor manufacturing a real object to include the created relief patternwith the critical surface texture; or selecting information first,followed by designing a suitable relief pattern to satisfy a criterion,for example, a degree of distortion in the information, and proceedingto creating an object or modifying an existing object to incorporate therelief pattern, finalizing the process by producing an encodedstructured illumination pattern. In principle, the object can be avirtual object, that is, created, defined or stored as a computer model.It can be used to encode and decode information in the virtualenvironments.

An example of the process with an alternate sequence of steps isillustrated in FIG. 6 as follows. In step 600 a desired relief patternis defined in the form of a model or created as a physical prototype asa part of a surface. Next, information is selected and represented in adesirable physical form in step 610. This physical representation canoptionally include verification of the suitability of itscharacteristics or parameters to be subjected to a sufficient distortionby the created relief pattern. Alternatively, it can be chosen byadherence to one or more pre-established rules, for example, byselecting a specific font size or color. In step 620 a target object isprovided and a portion of its surface is designated for the usage in theencoding process. This designated portion of the surface is thenmodified in step 630 to include or match the designed relief pattern ofstep 600. Subsequently, the encoded structured light pattern is createdin step 640, for example, using a similar method to the step 325 of FIG.3; and stored in the memory in the step 650 of FIG. 6.

Another embodiment with the alternate sequence of steps is shown in FIG.7, where the process starts with selecting information in step 700,similarly to the descriptions provided earlier. Subsequent steps includeselecting an object (step 710), designing a surface relief pattern (step720), creating and storing an encoded illumination pattern usingselected information and the designed surface relief texture or pattern(step 730), and modifying the object's surface to contain the designedsurface texture (step 740).

Optionally, during the process of encoding information in the structuredlight patterns illustrated in FIG. 3, FIG. 6 and FIG. 7, additionaliterations of the described steps can be further employed to betterdefine, refine and create these three elements: surface relief texture,information and encoded structured light pattern, which mutually serveas a triplet for the encoding and decoding information. Alternateembodiments can incorporate steps to test and verify the encoded lightpattern for unrecognizability and robustness against retrievinginformation without the appropriate key, which is served by the criticalsurface relief texture.

In yet another embodiment, the process of creating an encodedillumination pattern can be modified such, that after the step ofselecting information, the selected information is then scrambled ordistorted to satisfy a pre-selected criterion, for example, a thresholdamount of distortion to enable unrecognizability. Such distortion, canbe created, for example, using existing warping algorithms. Creation ofsuch distortion can be described as follows. The information representedin the form of an image is partitioned into local segments and a warpingalgorithm is separately applied to each of the local image segments toproduce different localized distortions for corresponding segments.Local distortions are used to avoid application of the global warpingtransformation, because globally applied distortion can be indentifiedusing mathematical analysis, reducing thus the robustness of theencoding. An amount of distortion is then evaluated to satisfy arecognizability criterion. If the amount of distortion is belowrecognizability threshold, or, inversely, above the unrecognizabilitythreshold, the distorted information is then used to generate theencoded structured light pattern using the inverse projection methoddescribed previously. The surface relief pattern is defined by theparameters of the local distortion used in the process of generatinglocally warped image with the information. Finally, the surface reliefpattern can be produced as a part of an object or implemented in acomputational form as a virtual object or model. Conversely, thedistorted information can be treated as an encoded information, and theinverse projection can be used to characterize the critical surfacerelief pattern, which is needed to compensate for the distortion in theprocess of retrieving information,

FIG. 8 illustrates a method for retrieving information encoded in astructured illumination pattern in accord with the present invention. Toretrieve information from its encoded form, an object with the criticalsurface incorporating the critical surface texture pattern or surfacerelief pattern, is provided in step 800. There can exist numerous waysof providing or obtaining such object including locating it, sending,manufacturing based on the provided specification, or transmitting viathe network or other methods of data transmission the datamathematically describing the object in the form of a model, orrepresenting the object in a virtual form including visualizing theobject using computer graphics means.

Next, in step 810 one or more portions containing critical surfacetexture or surface relief patterns are selected. The selection can beindicated in the specification or description of the object in anysuitable form, or, in other embodiments, separately communicated inrelation to the object, or the description of the object, or itselements. Alternatively, the marks can be made or identified on theobject as fiducials for the purpose of determining the critical surfacelocations.

Yet, in other embodiments the process of selection of the criticalsurface texture patterns can consist of repeated trials in the form of asearch, where in each step an arbitrary portion of the object's surfaceis selected or using a rule based search, thus selecting a number ofcandidate portions. In step 820 the encoded information in the form ofthe structured light pattern is provided. Similarly to obtaining theobject, the encoded information can be transmitted wirelessly or via thewired network, send as a physically printed pattern that can beconverted into a digital image to obtain the structured light patternusing for example, scanning technique, handed or located on a computerstorage media, or communicated in the form of a data that can be used toreconstructed the structured light pattern including the reconstructionas a computerized model. Then, the encoded structured light pattern isprojected onto the one or more selected portions of the surface in step830.

The projection can be utilized using various digital projection devicesor devices capable of emitting illumination in the form of an imagereproducing the specifications of the encoded light pattern.Alternatively, the projection can be simulated using computer graphicsprograms or in purely mathematical form without the visualization part.The projection can also include selecting certain parts of the encodedinformation that can be simultaneously or in succession projected ontoone or more selected surface relief patterns. If the encoded informationis represented as a series of frames, segments or as a digital videofile, a specific sequence of projecting can be followed which can bepre-established, separately communicated or defined using a rule.

In step 840 the detection of the reflected pattern or patterns thatresults from the projection takes place. The detection can be donevisually or using a capture device. If the projection is utilizedvirtually, the detection can also be done virtually, thus in the form ofthe computerized data created using the processor, which can be storedin the computer memory, and, optionally, visualized via displaying orprinting.

Finally, in step 850 the detected in step 840 pattern or patterns areprocessed to retrieve information. In one embodiment, the processing forretrieving information is performed visually. In other embodiments, itcan be done automatically, for example, using pattern recognitionalgorithms. Alternatively, the detected on step 840 pattern or patternscan be tested against an existing database of pattern, such that thedetected patterns can be compared with the patterns in the databaseusing various types of analysis techniques, including visual analysisand automatic analysis to identify information. The retrievedinformation including, the results of an automatic processing can bestored, transmitted for additional usage, including further analysis, orcommunicated as an end result.

FIG. 9 is a schematic of a system for retrieving information encoded ina structured illumination pattern in accord with the present invention.The digital projector 110 projects a structured illumination pattern 910with encoded information (in this case, a single character) onto thecritical surface texture 900. The structured illumination pattern isshown here as a warped version of a character to represent the fact thatit has been mapped through the inverse transformations X(u,v) and Y(u,v)as explained above. After reflection from the critical surface texture900, the encoded information 920 is captured by the digital camera 140for output.

In some configurations, the system for retrieving information encoded ina structured illumination pattern can be part of a remote encodedcommunication system. In one configuration, the sender of theinformation is in possession of the inverse transformation information,represented above by the inverse transformations X(u,v) and Y(u,v),while the receiver is in possession of the critical surface texture 900.The sender of the information can encode information by mapping adesired output image, the encoded information 920, through the inversetransformations X(u,v) and Y(u,v) to arrive at a structured illuminationpattern 910 with encoded information. The structured illuminationpattern 910 can be transmitted to the receiver for projection onto thecritical surface texture 900, reflection from the critical surfacetexture 900, and capture of the encoded information 920 by a digitalcamera. Transmission of the structured illumination pattern 910 from thesender to the receiver can be accomplished in a number of ways. In oneconfiguration, the structured illumination pattern 910 is a digitalimage that is compressed using standard image compression algorithms,such as the JPEG standard, transmitted to the receiver over theInternet, by CD-ROM or other hard storage media, or by USB-memorydevices, and then uncompressed to full size for projection. Thestructured illumination pattern 910 can also be digitally encryptedusing a public/private key encryption method for added security. Inother configurations, the structured illumination pattern 910 can bewritten onto hard copy media, physically delivered to the receiver, andthen digitally scanned into a computer for projection onto the criticalsurface texture 900.

The process of encoding and retrieving information using structuredillumination can be applied to authenticate an object having a 3Dtexture or a 3D relief pattern. FIG. 10 is a flow diagram of a methodfor authenticating an object using information encoded in a structuredillumination pattern, in accord with the present invention. The objectto be tested is provided in step 1000 and at least one identifyingportion is selected in step 1010. The identifying portion can be acritical surface texture or critical surface relief pattern, which cancontain unique surface characteristics of the object, differentiatingthe authentic (reference) object from similar objects. As explained inthe description of FIG. 8, the selection of the identifying portion canbe included in the specification or description of the object, or inother embodiments, separately communicated in relation to the object,the description of the object, or its elements. In addition, marks canbe made or engraved on the object as references or fiducials for thepurpose of determining the critical surface locations. In yet otherembodiments the selection of the identifying portion with the criticalsurface texture patterns can consist of repeated trials in the form of asearch, in which each step a candidate portion of the object's surfaceis selected, or, conversely, the search can be a rule based search, andlead to selecting a number of candidate portions. In step 1020, a firstreference pattern is selected and then projected onto the test object instep 1030 using a system such as the ones illustrated in FIG. 1 or FIG.2. This first reference pattern is an encoded structured light pattern,and is applied onto the object to test its authenticity. Such testpattern can be used together with the authentic object having a criticalsurface texture, as a lock and key pair to retrieve the information andthereby authenticate the object. The test pattern can be retrieved froma database of test patterns, or can be reconstructed from parametersstored in a database, or can be provided independently. The reflectionof the first reference pattern from the object 1050 is detected in step1040 This reflected pattern is then evaluated regarding the encodedinformation. This information can be contained in a second referencepattern. The second reference pattern, is then selected in step 1070,and any deviations between this pattern and the reflected pattern 1050are detected in a comparison step 1060. A decision is made in step 1080based on the deviations detected in step 1060 and a threshold. If thedeviations are below the threshold, then the object is authenticated instep 1100. If the deviations are above the threshold, then the object isrejected for authentication in step 1090.

The entire authentication process described in FIG. 10 can be performedautomatically, including step 1060, which detects deviations between thereflected pattern and the second reference pattern. Alternatively, allor some of the steps can be performed manually. The evaluation of thereflected pattern 1050 can also be based on visual inspection.Similarly, the comparison in step 1060 with the second reference patternand detections of any deviations can also be done visually. While insome embodiments it can be advantageous to have the second referencepattern available at the time and place of authentication, for example,displayed or otherwise represented, such as in the form of a print or ascanned document; in other embodiments the second reference pattern maybe stored remotely and the comparison process performed at anotherlocation and or delayed in time. In an alternative embodiment, thesecond reference pattern can be stored in the memory of the personperforming authentication, and the comparison in step 1060 consists ofdetermining if the reflected pattern matches memorized second referencepattern. The comparison in step 1060 can be based on matching physicalappearance of patterns, with the threshold applied in step 1080 to thedifferences in physical appearance (visually assessed or processed usingrelevant physical parameters, such as color, shape, edge information,etc.) to establish or reject the match, and therefore, confirm or rejectthe object as being authentic. Alternately, the comparison can beperformed based on the semantic interpretation of the patterns, althoughtheir physical appearance can be different. For example, both patternscan contain text messages or other types of symbols having similarmeaning but expressed differently. Therefore, the comparison involvesevaluating the similarity of the meaning or semantic interpretations. Inanother embodiment the mere acknowledgement of semantic or visualinterpretability of the reflected pattern 1050 can be used as acriterion of the authenticity (for example, in decision step 1080)stemming from the fact that the encoded information is successfullyretrieved if it is interpretable, and therefore the object is deemedauthentic in step 1100; or, conversely, the object is rejected as beingauthentic in step 1090, if the information cannot be interpreted,visually or semantically from the reflected pattern 1050. In suchembodiment steps 1060 and 1070 can be omitted.

FIG. 11 is a flow diagram of a method for authentication ofthree-dimensional structures, in accord with the present invention. Instep 1110, a first pattern of illumination is projected onto thethree-dimensional structure using a system such as those illustrated inFIG. 1 or FIG. 2. The patterns of illumination can be retrieved from adatabase of illumination patterns, or can be reconstructed fromparameters stored in a database, or can be provided independently. Next,the reflection of the first illumination pattern from thethree-dimensional structure is detected in step 1120. Referring to FIG.1, detection of the first illumination pattern includes forming an imageof the pattern via the digital camera lens 205 onto the image sensor215. The first reflected pattern is then analyzed for a firstcharacteristic in step 1130.

Such characteristics include well-known first order statistics such asmean, standard deviation, median, or the grey level histogram in one ormore color channels. Characteristics can also include, but are notlimited to, second or higher order statistics such as theautocorrelation function, power spectrum, or kurtosis in one or morecolor channels. In some configurations, characteristics can include theresults of other linear, nonlinear or morphological image processingoperations applied to one or more color channels, information theoreticmetrics, or metrics pertaining to possible correlations between colorchannels. In yet another configuration, analyzing for specificcharacteristics can include comparing the first reflected pattern to adatabase of patterns. Such a comparison can include searching for amatching pattern, searching for a pattern that has matching derivedstatistics, or searching for a pattern that has a mathematicalcorrelation exceeding a threshold.

In the next step 1140, the first reflected pattern is inverted, usingthe mathematical inversion operations described previously. The invertedpattern is then projected 1150 onto the three-dimensional structure, andthe reflection of this pattern, which will be termed the first invertedreflected pattern, is detected 1160. This detected pattern is thenanalyzed 1170 for a second characteristic. The characteristics includethose mentioned above, and also characteristics related to differencesbetween the first and second detected patterns. A decision step 1180then determines whether the first and second characteristics pertainingto an authentic object are present. If both characteristics are present,then the object is authenticated in step 1190. If either characteristicis not present, then the object is rejected for authentication in step1200.

This method of authentication relies on examination of a firststructured light pattern that is distorted by the surface. Thisreflected pattern is analyzed to determine if the distortions presentare those which are characteristic of the authentic object. As anadditional test, the detected pattern is inverted and projected backonto the object. If the object is authentic, this step will result in anew reflected pattern (the first inverted reflected pattern) that shouldmatch the original, first structured light pattern. If the object is notauthentic, the inverted image contains errors with respect to anauthentic inverted image. Projection of an erroneous inverted image ontothe object results in further errors that enhance the mismatch betweenthe characteristics of the final detected image and the original firststructured light pattern. In another configuration, the first invertedreflected pattern can be directly compared to the original projectedstructured lighting pattern, rather than comparing derivedcharacteristics or statistics from each pattern.

FIG. 12 is a flow diagram of a method for authentication ofthree-dimensional structures, in accord with the present invention. Instep 1210, a first pattern of illumination is projected onto thethree-dimensional structure using a system such as those illustrated inFIG. 1 or FIG. 2. The patterns of illumination can be retrieved from adatabase of illumination patterns, or can be reconstructed fromparameters stored in a database, or can be provided independently. Next,the reflection of the first illumination pattern from thethree-dimensional structure is detected in step 1220. Referring to FIG.1, detection of the first illumination pattern includes forming an imageof the pattern via the digital camera lens 205 onto the image sensor215. The first reflected pattern is then analyzed for a firstcharacteristic in step 1230. Such characteristics include thosedescribed above in connection with FIG. 11.

In the next step 1240, a second pattern of illumination is projectedonto the three-dimensional structure, similar to the projection of thefirst pattern of illumination. Next, the reflection of the secondillumination pattern from the three-dimensional structure is detected instep 1250. The second reflected pattern is then analyzed for a secondcharacteristic in step 1260, similar to the analysis of the firstreflected pattern. A decision step 1270 then determines whether firstand second characteristics pertaining to an authentic object arepresent. If both characteristics are present, then the object isauthenticated in step 1280. If either characteristic is not present,then the object is rejected for authentication in step 1290.

FIG. 13 is a flow diagram of an alternative method for authentication ofthree-dimensional structures, in accord with the present invention. Instep 1310, a first pattern of illumination is projected onto thethree-dimensional structure using a system such as those illustrated inFIG. 1 or FIG. 2. Once again, single or multiple patterns ofillumination can be retrieved from a database, or can be reconstructedfrom parameters stored in a database, or can be provided independently.Next, the reflection of the first illumination pattern from thethree-dimensional structure is detected in step 1320. Referring to FIG.1, detection of the first illumination pattern includes forming an imageof the pattern via the digital camera lens 205 onto the image sensor215. The first reflected pattern is then analyzed for a firstcharacteristic in step 1230. Such characteristics include thosedescribed above in connection with FIG. 11

The detected pattern is then analyzed 1340 to detect deviations from areference pattern 1330. A decision step 1350 then determines whether thedeviation is below a threshold. If the deviation is below threshold,then the object is authenticated in step 1360. If the deviation is notbelow threshold, then the object is rejected for authentication in step1370.

It will be appreciated by those skilled in the art that these methodscan be extended and elaborated within the scope of the invention. Forexample, the methods of FIGS. 10-13 can be extended to compare areference object with a test object as follows: project a first patternof illumination onto a reference three-dimensional structure, detectingand recording the reflection of this pattern from the referencestructure; invert the image of the reflected pattern; create a secondpattern of illumination from the inverted image; project this secondpattern onto a structure to be authenticated; compare the reflectedpattern of light from the object under test to the first pattern ofillumination; and authenticate the structure if the measured deviationsare within a threshold. In one configuration, the party performing theauthentication is only in possession of the parameters of the system forilluminating the reference and test objects, the inversion algorithm,which can be in the form of executable code, and the first pattern ofillumination. The sender of the information can supply the referenceobject and the code for the inversion algorithm, as well as the firstpattern of illumination

Also within the scope of the invention is the authentication ofthree-dimensional structures based on their spectral or colorcharacteristics. In particular, a method for authentication ofthree-dimensional structures can include capturing a multicolor image ofa reference structure, where a multicolor image is understood to be acapture using more than one spectral sensitivity or spectral bandpass.Image captures of multiple spectral bandpass can be generated throughthe use of multiple color filters or other wavelength-dependentcomponents inserted into the optical path of a capture device, orthrough the use of multiple capture devices.

Following the capture of the multicolor image, the method continues withthe creation of at least one pattern of illumination based on at leastone color of the reference structure. Basing the pattern of illuminationon at least one color of the reference structure includes selecting atleast one spectral band wherein the reference structure has significantspectral reflectance values, or selecting at least one region orcritical surface texture that exhibits this same property. The criticalsurface texture selected may or may not include additional spectralbands of significant spectral reflectance.

In one configuration, the pattern of illumination created can includeinversion of a pattern that illuminates the reference object during thecapture of the multicolor image. Next, the created illumination patternis projected on the three-dimensional structure to be authenticated, andthe reflected pattern is analyzed for a pre-determined characteristic.As described previously with reference to other configurations of theinvention, the pre-determined characteristic can be provided along withor separately from the structure to be authenticated, or can beretrieved from a database. The structure is then authenticated if thepre-determined characteristic is present. In one configuration, thepre-determined characteristic is a color difference at, above or below aparticular threshold value. In another configuration, the pre-determinedcharacteristic is both a color characteristic and a geometric portion orproperty found in the reference structure.

FIG. 14 is a flow diagram of another embodiment of the method forauthenticating an object using information encoded in a structuredillumination pattern, in accord with the present invention. The objectto be tested is provided in step 1410 and an identifying portion of thetest object having a surface with the surface texture or surface reliefis selected in step 1420. The identifying portion of the test objectsurface as mentioned above can contain critical surface texture orcritical surface relief pattern, which can contain unique surfacecharacteristics of the reference authentic object. In other words, theidentifying portion of the test object can be expected to containcritical surface texture, relief pattern or critical surfacecharacteristics, differentiating the authentic and/or reference objectfrom similar objects, if the test object is authentic. In step 1430, apattern of illumination is obtained. The pattern of illumination can beretrieved from a database of test patterns, or can be reconstructed fromparameters stored in a database, from a scanned document or a printedimage, or can be provided independently. This pattern is the encodedstructured illumination pattern, or the reference pattern ofillumination, specifically designed or adapted to the identifyingportion of the surface or surface relief of the reference object. Themethod for designing, creating or adapting the encoded illuminationpattern to the identifying portion of the surface with critical surfacetexture or surface relief pattern has been illustrated earlier in FIG.3, FIG. 6 or FIG. 7. The encoded structured illumination pattern is thenprojected onto the identifying portion of the test object surface instep 1440 using a system such as the ones illustrated in FIG. 1 or FIG.2. The reflection of the projected pattern or reflected pattern from theobject 1460 is detected in step 1450. The detection can be done using anappropriate image capture device, optical sensors, including capable ofcapturing spectral information indicative of material properties andgeometric properties, or can be done visually, in the form of observingthe pattern. This reflected pattern is processed in step 1470. Theprocessing can involve analyzing the pattern using visual assessment,automatic assessment, applying one or more image processing, featureextraction, computer vision, object, character recognition, spectral orsemantic analysis algorithms or other appropriate manual or automaticprocessing techniques aiming at retrieving, decoding or decipheringinformation that can be contained in the reflected pattern, includinglinking it to the codes, ciphers, tables or databases. As it wasdescribed earlier, the processing can also include registering andanalyzing elements of physical appearance, and physical parameters orcharacteristics influencing such physical appearance, visually orautomatically extracted, or registering and analyzing semantic meaningor semantic or visual interpretability of the reflected pattern. Next,in step 1480 the processed reflected pattern is evaluated or tested todetermine if it contains encoded information. The determination caninclude comparing the processed and analyzed pattern and theextracted/retrieved information with the expected pattern and orexpected information. The expected encoded information can be obtainedfrom the database, via communication or through other means describedpreviously and can be expressed in the form of a pattern, a text,message, geometrical figures, symbols, codes, numbers or mathematical orcomputer models, formulas etc. With respect to the embodiment of FIG.14, such pattern with the expected information can be called a referenceinformation pattern. In the context of FIG. 10 such pattern is calledthe second reference pattern.

The comparison can utilize applying various measures, for example,interpretability, legibility, presence of specific colors, shapes, andcodes, specific meaning, number of separate elements and otherappropriate criteria. The comparison can also be based on determiningthe deviations between the processed pattern and expected pattern, orbetween extracted information from the reflected pattern as a result ofprocessing and expected information, expressing the deviations in theform of a measure. A decision is made in step 1490 based on theevaluation and a criterion that is used to determine if the processedreflected pattern contains encoded information, for example, a thresholdwith respect to detected deviations between the processed reflectedpattern and the pattern containing expected information. If the utilizedcriterion is satisfied the object is authenticated in step 1491.Conversely, if the criterion is not satisfied, then the object isrejected for authentication in step 1492.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the scope of theinvention. The steps of the invention may be performed in an orderdifferent from that specified in the claims and still fall within thescope of the invention.

PARTS LIST

-   110 projector-   115 object-   120 illumination pattern-   120A illumination pattern-   120B illumination pattern-   130 relief pattern (textured surface)-   135 selected portion-   140 camera-   140A camera-   140B camera-   205 capture lens-   205A capture lens-   205B capture lens-   210 projection lens-   215 image sensor-   215A image sensor-   215B image sensor-   220 light modulator-   225 illumination system-   226 baseline-   230 processor-   245 interface system-   250 processor-accessible memory system-   310 surface relief of object characterized-   312 select information-   315 projection direction-   320 select portion of object with relief pattern-   325 create encoded illumination path-   335 capture direction-   400 substrate-   405 relief element-   410 relief element-   415 relief element-   420 binary image representation-   425 white pixels-   430 black pixels-   500 surface texture-   505 row-   510 row-   515 row-   520 row-   525 row-   600 create relief pattern-   610 select information-   620 object and select portion provided with native surface relief    pattern-   630 modify selected portion-   640 create structured light pattern-   650 store reference structured light pattern-   700 select information-   710 select object-   720 design surface relief pattern-   730 create and store encoded illumination pattern-   740 modify object surface-   800 provide object-   810 select portion of surface-   820 provide structured light pattern-   830 project structured light pattern-   840 detect reflected pattern-   850 process detected pattern-   900 surface texture-   910 illumination pattern-   920 encoded information-   1000 provide test object-   1010 select identifying portion-   1020 select first reference pattern-   1030 project first reference pattern-   1040 detect reflected pattern-   1050 reflected pattern-   1060 detect deviation from reference-   1070 select second reference-   1080 below threshold pattern-   1090 reject-   1100 authenticate-   1110 project first pattern-   1120 detect first reflected pattern-   1130 analyze first reflected pattern for first characteristic-   1140 invert first reflected pattern-   1150 project first inverted pattern-   1160 detect first inverted reflected pattern-   1170 analyze first inverted reflected pattern for second    characteristic-   1180 first and second characteristics present-   1190 authenticate-   1200 reject-   1210 project first pattern-   1220 detect first reflected pattern-   1230 analyze first reflected pattern for first characteristic-   1240 project second pattern-   1250 detect second reflected pattern-   1260 analyze second reflected pattern for second characteristic-   1270 first and second characteristics present-   1280 authenticate-   1290 reject-   1310 project first pattern-   1320 detect first reflected pattern-   1330 reflected pattern-   1340 detect deviation from reference-   1350 deviation below threshold-   1360 authenticate-   1370 reject-   1410 provide test object-   1420 select identifying portion-   1430 obtain encoded illumination pattern-   1440 project illumination pattern-   1450 detect reflected pattern-   1460 reflected pattern-   1470 process reflected pattern-   1480 evaluate reflected pattern-   1490 contain information-   1491 authenticate-   1492 reject

1. A method for encoding information in a structured illuminationpattern comprising: a) selecting information to be encoded based on thegeometrical and material properties of a surface relief pattern; b)characterizing a relief pattern of an object's surface; c) selecting atleast one portion of the object's surface including the surface reliefpattern, based on the amount of distortion introduced by the portion ofthe surface relief pattern to the information being encoded; d) creatingat least one structured illumination pattern responsive to the selectedsurface relief pattern and the selected information; and e) storing thestructured illumination pattern for subsequent decoding.
 2. The methodof claim 1 wherein creating a structured illumination pattern includesan inversion.
 3. The method of claim 1 wherein the structuredillumination pattern is monochromatic.
 4. The method of claim 1 whereinthe structured illumination pattern is polychromatic.
 5. The method ofclaim 1 wherein the structured illumination pattern stimulates emissionof radiation from the object's surface.
 6. The method of claim 1 whereinthe structured illumination pattern is a series of patterns.
 7. Themethod of claim 1 wherein the stored structured illumination patternscomprise a video sequence. 8-30. (canceled)